System and method for tracking multiple objects

ABSTRACT

The present invention relates to a system for tracking an object. The system includes an image capturing unit configured to capture a video of a predetermined observation area and output the captured video; and a multi-object tracker configured to output an object-tracking image by tracking multiple objects within an object image which is generated by extracting the objects from each of image frames obtained from the video obtained from the image capturing unit, wherein the multi-object tracker determines whether occlusion of the objects or hijacking occurs while performing multi-object tracking, and when it is determined that at least one of the occlusion and hijacking occurs, the multi-object tracker outputs the object-tracking image corrected by removing the occurring occlusion or hijacking.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to and the benefit of Korean PatentApplication No. 10-2017-0019427, filed on Feb. 13, 2017, the disclosureof which is incorporated herein by reference in its entirety.

BACKGROUND 1. Field of the Invention

The present invention relates to an object tracking system, and moreparticularly, to a system and method for tracking multiple objects.

2. Discussion of Related Art

In the sports and security service fields, a service capable ofaccurately tracking multiple target objects (e.g., people) and providinginformation about the objects by analyzing images obtained throughcamera tracking on the target objects is provided.

In the case where multiple target objects are tracked to provide theinformation about the objects, there may be problems of occlusion amongobjects and hijacking which causes a target object to be changed (i.e.,causes another object to be tracked) due to the similarities between theobjects.

Therefore, there is a need for an approach that prevents an objecttracking error due to occlusion of objects and hijacking that tracksanother target object.

SUMMARY OF THE INVENTION

Therefore, the present invention is devised to solve the aforementionedproblems, and the objective of the present invention is to provide asystem and method for tracking multiple objects, which is capable ofproviding accurate information about a tracked object by handlingocclusion of objects and hijacking which may occur when multiple objectsare tracked simultaneously.

In one general aspect, there is provided a system for tracking multipleobjects including: an image capturing unit configured to capture a videoof a predetermined observation area and output the captured video; and amulti-object tracker configured to output an object-tracking image bytracking multiple objects within an object image which is generated byextracting the objects from each of image frames obtained from the videoobtained from the image capturing unit, wherein the multi-object trackerdetermines whether occlusion of the objects or hijacking occurs whileperforming multi-object tracking, and when it is determined that atleast one of the occlusion and the hijacking occurs, the multi-objecttracker outputs the object-tracking image corrected by removing theoccurring occlusion or hijacking.

The multi-object tracker may output a current object-tracking image whenit is determined that the occlusion of the objects and the hijacking donot occur.

When a bounding box of a target object is detected to be occluded by abounding box of another object, the multi-object tracker may determinethat the occlusion of the objects occurs.

When it is determined that the occlusion of the objects occurs, themulti-object tracker may estimate a bounding box and depth of each ofthe objects and remove objects except for a target object from theobject image.

The multi-object tracker may determine a depth order of the objects onthe basis of the estimated bounding boxes and depths of the objects,recognize at least one of objects in front of and behind the targetobject on the basis of the determined depth order of the objects, andremove the recognized object.

When a displacement of a bounding box of a target object is detected tobe identical to a displacement of a bounding box of another object for apredetermined time period, the multi-object tracker may determine thatthe hijacking occurs.

When it is determined that the hijacking occurs, the multi-objecttracker may remove an object being actually tracked by a bounding box ofa target object from the object image so that the bounding box tracksthe target object.

The multi-object tracker may generate a reference background image bymodeling a reference background using a reference background modelingvideo obtained from the image capturing unit and generate the objectimage by extracting the objects from each of the image frames on thebasis of comparison between the reference background image and each ofthe image frames obtained from a video for object tracking obtained fromthe image capturing unit.

The multi-object tracker may obtain a color difference by comparingcolors of the reference background image and each of the image frames,and extract the objects from each of the image frames on the basis ofthe obtained color difference.

When the generated object image is an initial object image, themulti-object tracker may allocate a bounding box to each of the objectsto initialize multi-object tracking and perform the multi-objecttracking.

In another general aspect, there is provided a method of trackingmultiple objects including: generating a reference background image bymodeling a reference background using a reference background modelingvideo obtained from an image capturing unit; generating an object imageby extracting objects from each of image frames on the basis ofcomparison between the reference background image and each of the imageframes obtained from a video for object tracking obtained from the imagecapturing unit; and outputting a current object-tracking image or acorrected object-tracking image according to a result of determinationon whether occlusion of the objects or hijacking occurs, whileperforming multi-object tracking on the basis of the object image.

Whether the occlusion of the objects occurs may be determined on thebasis of detection of whether a bounding box of a target object isoccluded by a bounding box of another object.

Whether the hijacking occurs may be determined on the basis of detectionof whether a displacement of a bounding box of a target object isidentical to a displacement of a bounding box of another object.

The outputting of the current object-tracking image may be performedwhen it is determined that the occlusion of the objects and thehijacking do not occur.

When it is determined that the occlusion of the objects occurs, theoutputting of the corrected object-tracking image may be performedwherein the object-tracking image may be corrected by removing objectsexcept for a target object from the object image by estimating abounding box and depth of each of the objects.

When it is determined that the hijacking occurs, the outputting of thecorrected object-tracking image may be performed wherein theobject-tracking image may be corrected by removing an object beingactually tracked by a bounding box of a target object from the objectimage so that the bounding box tracks the target object.

When it is determined that the occlusion of the objects and thehijacking occur, the outputting of the corrected object-tracking imagemay be performed wherein the object-tracking image may be corrected byremoving objects except for a target object from the object image byestimating a bounding box and depth of each of the objects and iscorrected also by removing an object being actually tracked by abounding box of the target object from the object image so that thebounding box tracks the target object.

The removing of the object other than the target object from the objectimage by estimating the bounding box and depth of each of the objectsmay include determining a depth order of the objects on the basis of theestimated bounding boxes and depths of the objects, recognizing at leastone of objects in front of and behind the target object on the basis ofthe determined depth order, and removing the recognized object.

The generating of the object image may include obtaining a colordifference by comparing colors of the reference background image andeach of the image frames and extracting the objects from each of theimage frames on the basis of the obtained color difference.

When initial multi-object tracking is performed, the performing of themulti-object tracking on the basis of the object image may includeallocating a bounding box to each of the objects to initialize themulti-object tracking and then performing the multi-object tracking.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects, features and advantages of the presentinvention will become more apparent to those of ordinary skill in theart by describing exemplary embodiments thereof in detail with referenceto the accompanying drawings, in which:

FIG. 1 is a diagram illustrating a configuration of a system fortracking multiple objects according to an exemplary embodiment of thepresent invention;

FIG. 2 is a flowchart illustrating operations of the system for trackingmultiple objects according to the exemplary embodiment of the presentinvention;

FIGS. 3A and 3B are flowcharts illustrating an operation of outputtingan object-tracking image performed by the system for tracking multipleobjects according to the exemplary embodiment of the present inventionin detail;

FIG. 4A is a picture showing an example of a video input to amulti-object tracker in one embodiment of the present invention;

FIG. 4B is a picture showing an example of a reference background imagemodeled by the multi-object tracker in the embodiment of the presentinvention;

FIG. 4C is a picture showing an example of an object image extractedfrom the input video by the multi-object tracker in the embodiment ofthe present invention;

FIG. 5 is a picture for describing a method of determining whetherhijacking occurs according to an embodiment of the present invention;

FIG. 6A is a picture showing an example of an original video input tothe multi-object tracker in the embodiment of the present invention;

FIG. 6B is a picture showing an image in which hijacking occurs in theembodiment of the present invention;

FIG. 6C is a picture showing an image from which the hijacking isremoved in the embodiment of the present invention;

FIG. 7A shows an example in which occlusion of objects occurs in theembodiment of the present invention;

FIG. 7B shows an example in which the occlusion is removed in theembodiment of the present invention;

FIG. 8 is a table for comparing center location errors of the system fortracking multiple objects according to the embodiment of the presentinvention and a conventional object tracking system;

FIG. 9 is a table for comparing success rates of the system for trackingmultiple objects according to the embodiment of the present inventionand the conventional object tracking system; and

FIG. 10 is a table for comparing multi-object tracking times of thesystem for tracking multiple objects according to the embodiment of thepresent invention and the conventional object tracking system.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

The particular structural or functional descriptions of embodimentsaccording to the concepts of the present invention disclosed in thespecification or the application are only intended for the purpose ofdescribing embodiments according to the concepts of the presentinvention and the embodiments according to the concepts of the presentinvention may be practiced in various forms and should not be construedas being limited to those described in the specification or theapplication.

While the present invention is susceptible to various modifications andalternative forms, specific embodiments thereof are shown by way ofexample in the drawings and will herein be described in detail. Itshould be understood, however, that there is no intent to limit thepresent invention to the particular forms disclosed, but on thecontrary, the present invention is to cover all modifications,equivalents, and alternatives falling within the spirit and scope of thepresent invention as defined by the claims.

Although the terms first, second, etc. may be used herein to describevarious components, these components should not be limited by theseterms. The terms may be only used for the purpose of distinguishing onecomponent from another component, e.g., a first component may be namedas a second component without departing from the scope of the presentinvention according to the concepts of the present invention andsimilarly, a second component may also be named as a first component.

When any component is referred to as being “connected” to anothercomponent, it should be understood that the former can be “directlyconnected” to the latter, or intervening components may be present. Incontrast, when a component is referred to as being “directly connected”to another component, it should be understood that there are nointervening components. Other expressions describing the relationshipbetween components, such as “between” and “directly between” or“adjacent to” and “adjacent directly to” should be also construed in thesame way.

The terms used herein are only used to describe specific embodiments andnot intended to limit the present invention. In the followingembodiments, the terms in singular form may include the plural formunless otherwise specified. It should be understood that the terms“includes” and/or “including,” or “comprises” and/or “comprising”indicate the presence of characteristics, numbers, steps, operations,components, parts or combinations thereof represented in the presentdisclosure but do not exclude the presence or addition of one or moreother characteristics, numbers, steps, operations, components, parts orcombinations thereof.

Unless otherwise defined herein, all terms used herein includingtechnical or scientific terms have the same meanings as those generallyunderstood by a person skilled in the art. Terms defined in dictionariesgenerally used should be construed to have meanings matching withcontextual meanings in the related art and are not construed as an idealor excessively formal meaning unless otherwise defined herein.

It should also be noted that, in some alternative implementations, thefunctions noted in the block may occur out of the order noted in thefigures. For example, two blocks shown in succession may, in fact, beexecuted substantially concurrently, or the blocks may sometimes beexecuted in the reverse order, depending upon the functionalityinvolved.

Hereinafter, a system and method for tracking multiple objects accordingto the present invention will be described in detail with reference tothe accompanying drawings.

FIG. 1 is a diagram illustrating a configuration of a system fortracking multiple objects according to an exemplary embodiment of thepresent invention.

Referring to FIG. 1, a system 100 for tracking multiple objects(hereinafter, will be referred to as a “system”) according to theexemplary embodiment of the present invention handles occlusion ofobjects and hijacking, which occur when multiple objects aresimultaneously tracked and provides accurate information about a targetobject.

To this end, the system 100 includes at least one processor which tracksmultiple objects while handling occlusion of objects and hijacking, andat least one memory configured to store necessary information (e.g., amulti-object tracking algorithm, object-occlusion handling algorithm, ahijacking handling algorithm, etc.) for tracking the multiple objectswhile handling the occlusion of objects and the hijacking.

Specifically, the system 100 may include an image capturing unit 110, astorage unit 130, a multi-object tracker 150, and a display 170.

The image capturing unit 110 is installed to capture a video of apredetermined area of interest to be observed (e.g., inside of astadium, a security surveillance area, etc.) and provides a capturedvideo to the multi-object tracker 150.

In the present invention, the video obtained by the image capturing unit110 is used for modeling a reference background or tracking multipleobjects.

For description, a video used for modeling the reference background willbe referred to as a “reference background modeling video” and an imageused for tracking multiple objects will be referred to as a “video forobject tracking.

The storage unit 130 may be configured with at least one memory andconfigured to store necessary information (e.g., a multi-object trackingalgorithm, an object-occlusion handling algorithm, a hijacking handlingalgorithm, etc.) for the multi-object tracker 150 to track multipleobjects.

The multi-object tracker 150 tracks target objects within the videoprovided from the image capturing unit 110 and provides anobject-tracking video to the display 170.

In this case, the multi-object tracker 150 determines whether occlusionof objects or hijacking occurs. When it is determined that the occlusionof objects or the hijacking occurs, the multi-object tracker 150corrects the object-tracking video by handling the occlusion of objectsor the hijacking and provides the corrected object-tracking image to thedisplay 170.

Detailed functions related to the multi-object tracker 150 will bedescribed below.

The display 170 may display the information provided from themulti-object tracker 150, i.e., the image-tracking video, and may beconfigured to include a liquid crystal display (LCD) panel or an organiclight-emitting diode (OLED) panel.

In detail, the multi-object tracker 150 generates a reference backgroundimage by modeling a reference background using the reference backgroundmodeling video obtained from the image capturing unit 110.

In this case, the multi-object tracker 150 models the referencebackground on the basis of a background image in the referencebackground modeling video.

In addition, the multi-object tracker 150 extracts objects from eachimage frame on the basis of comparison between the reference backgroundimage and each image frame obtained from the video for object trackingobtained from the image capturing unit 110.

In this case, the multi-object tracker 150 obtains a color difference bycomparing colors of the reference background image and each image frame,extracts objects from each image frame on the basis of the obtainedcolor difference and generates an object image.

Then, the multi-object tracker 150 performs multi-object tracking on thebasis of the generated object image and provides the object-trackingimage to the display 170.

In this case, when the generated object image is an initial object image(i.e., an initial attempt for object tracking), the multi-object tracker150 initializes multi-object tracking by allocating a bounding box (atracking area) to each of the objects and then performs multi-objecttracking.

Also, in order to prevent a bounding box error from occurring in theprocess of tracking multiple objects, the multi-object tracker 150performs the multi-object tracking while allowing the bounding box totrack the target object using a difference image between an input imageand the background image.

In this case, the multi-object tracker 150 performs the multi-objecttracking while slightly moving the bounding box to around the targetobject so that the bounding box can fully contain the target objecttherein.

Meanwhile, the multi-object tracker 150 may perform the multi-objecttracking on the basis of a kernalized correlation filter (KCF) which isextended to track multiple objects.

A KCF-based multi-object tracking method is a tracking method thatrepeatedly performs a conventional KCF-based tracking method whichtracks a single object as many times as the number of target objects tobe tracked.

An algorithm for tracking multiple objects on the basis of a KCF whichis extended to track multiple objects may be expressed as follows:

for (i=1; i < the number of target objects to be tracked; i++) {  Single object KCF tracking }

In addition, the multi-object tracker 150 determines whether occlusionof objects or hijacking occurs, while performing multi-object tracking.When it is determined that the occlusion of objects or the hijackingoccurs, the multi-object tracker 150 generates a correctedobject-tracking image by handling the occlusion of objects or hijackingand provides the corrected object-tracking image to the display 170.

In this case, when a bounding box of the target object is occluded by abounding box of another object, the multi-object tracker 150 determinesthat occlusion between the objects occurs.

For example, when objects of similar colors occlude each other, trackingof a target object may fail due to an object with a similar appearancein front of and/or behind the target object.

When it is determined that the occlusion of objects occurs, themulti-object tracker 150 estimates a bounding box and depth of each ofthe objects within the object image and removes an object other than thetarget objects.

In this case, the multi-object tracker 150 determines a depth order ofobjects on the basis of the estimated bounding box and depth of eachobject, recognizes an object in front of and/or behind the target objecton the basis of the determined depth order, and removes the recognizedobject.

Particularly, the multi-object tracker 150 determines the order ofobjects on the basis of an object positioned at a top position in theobject image and determines that an object positioned upper than otherobjects is positioned farther away from the others.

In addition, the multi-object tracker 150 generates a correctedobject-tracking image by removing objects except for the target objectfrom the object image and provides the corrected object-tracking imageto the display 170.

Meanwhile, when a displacement for a predetermined time period (e.g., 10frames) set for the bounding box of the target object is detected to beidentical to a displacement for a predetermined time period (e.g., 10frames) set for a bounding box of another object, the multi-objecttracker 150 determines that hijacking occurs.

In other words, when the displacement of the bounding box of the targetobject is detected to be identical to the displacement of the boundingbox of another object for the predetermined time period, themulti-object tracker 150 determines that hijacking occurs.

For example, when a first bounding box intended for object A tracksobject B, the first bounding box moves to the same displacement as asecond bounding box that tracks object B, and thus based on thisphenomenon, whether the hijacking has occurred may be determined.

When it is determined that the hijacking has occurred, the multi-objecttracker 150 removes an object actually being tracked by the bounding boxof the target object (that is, an object which is not a target objectbut is being tracked) from the object image so as to generate acorrected object-tracking image such that the bounding box tracks thetarget object. Then, the multi-object tracker 150 provides the correctedobject-tracking image to the display 170.

Particularly, the multi-object tracker 150 removes objects being trackedby two bounding boxes, and may remove, for example, the common area ofthe two bounding boxes from the object image.

In the foregoing, the configuration and functions of the system fortracking multiple objects according to the exemplary embodiment of thepresent invention have been described. Hereinafter, a method of trackingmultiple objects in the system for tracking multiple objects accordingto the exemplary embodiment of the present invention will be described.

FIG. 2 is a flowchart illustrating operations of the system for trackingmultiple objects according to the exemplary embodiment of the presentinvention.

The operations shown in FIG. 2 may be performed by the system 100 whichis described with reference to FIG. 1. The multi-object tracker 150generates a reference background image by modeling a referencebackground using a reference background modeling video obtained from theimage capturing unit 110 (S200).

In operation S200, the multi-object tracker 150 models a referencebackground on the basis of a background image within the referencebackground modeling video.

After operation S200, the multi-object tracker 150 generates an objectimage by extracting objects from each image frame on the basis ofcomparison between the reference background image and each image frameobtained from a video for object tracking obtained from the imagecapturing unit 110 (S210).

In operation S210, the multi-object tracker 150 obtains a colordifference by comparing colors of the reference background image andeach image frame, extracts objects from each image frame on the basis ofthe obtained color difference and generates an object image.

FIG. 4A is a picture showing an example of a video input to amulti-object tracker in one embodiment of the present invention, FIG. 4Bis a picture showing an example of a reference background image modeledby the multi-object tracker in the embodiment of the present invention,and FIG. 4C is a picture showing an example of an object image extractedfrom the input video by the multi-object tracker in the embodiment ofthe present invention.

After operation S210, the multi-object tracker 150 determines whetherocclusion of objects or hijacking occurs, while performing multi-objecttracking (S220). According to a result of determination on whetherocclusion of objects or hijacking occurs, the multi-object tracker 150outputs a current object-tracking image or a corrected object-trackingimage through the display 170 (S230).

In this case, in operation S220, when an initial attempt formulti-object tracking is made, the multi-object tracking is initializedby allocating an object tracking area (a bounding box) to each of theobjects and then the multi-object tracking is performed.

In addition, in operation S220, the multi-object tracking may beperformed based on a KCF which is extended to track multiple objects.

Details of operation S230 will be described below with reference to FIG.3.

After operation S230, the multi-object tracker 150 determines whether anoperation termination is requested (S240). When the operationtermination is not requested (S240-NO), operation S210 is performed, andwhen the operation termination is requested (S240-YES), the operation isended.

FIGS. 3A and 3B are flowcharts illustrating the operation of outputtingan object-tracking image performed by the system for tracking multipleobjects according to the exemplary embodiment of the present inventionin detail.

The flowcharts shown in FIGS. 3A and 3B depict operation S230 of FIG. 2in detail. The multi-object tracker 150 determines whether occlusion ofobjects occurs (S300).

In operation S300, the multi-object tracker 150 determines whetherocclusion of objects occurs on the basis of a detection of whether thebounding box of each target object is occluded by a bounding box ofanother object.

In other words, in operation S300, when the bounding box of the targetobject is detected to be occluded by the bounding box of another object,the multi-object tracker 150 determines that the occlusion between theobjects occurs.

When it is determined in operation S300 that the occlusion of objectsdoes not occur (S300-NO), the multi-object tracker 150 determineswhether hijacking occurs (S310).

In this case, in operation S310, when a displacement of the bounding boxof the target object is detected to be identical to a displacement ofthe bounding box of another object for a predetermined time period(e.g., a predetermined number of frames), the multi-object tracker 150determines that hijacking occurs.

FIG. 5 is a picture for describing a method of determining whetherhijacking occurs according to an embodiment of the present invention.

Here, FIG. 5A is a picture showing an image frame at arbitrary time t,FIG. 5B is a picture showing an image frame at time (t+1), FIG. 5C is apicture showing an image frame at time (t+2), and FIG. 5D is a pictureshowing an image frame at time (t+3).

As shown in FIG. 5, when a displacement of a bounding box indicated by adashed line is detected to be identical to a displacement of a boundingbox indicated by a solid line for a predetermined number of frames, themulti-object tracker 150 determines that hijacking occurs.

When it is determined in operation S310 that hijacking does not occur(S310-NO), the multi-object tracker 150 outputs a currentobject-tracking image (S320).

When it is determined in operation S310 that hijacking occurs(S310-YES), the multi-object tracker 150 removes the hijacking (S330)and outputs an object-tracking image corrected through the removal ofhijacking (S340).

In operation S330, the multi-object tracker 150 removes an object beingactually tracked by the bounding box of the target object from theobject image such that the bounding box tracks the target object, andthereby the hijacking is removed.

FIG. 6 is a picture showing an example of removal of hijacking in theembodiment of the present invention. FIG. 6A is a picture showing anexample of an original video input to the multi-object tracker in theembodiment of the present invention, FIG. 6B is a picture showing animage in which hijacking occurs in the embodiment of the presentinvention, and FIG. 6C is a picture showing an image from which thehijacking is removed in the embodiment of the present invention.

FIG. 6 shows an example in which a bounding box indicated by a dashedline is tracking a non-target object (invalid object). When themulti-object tracker 150 removes the hijacking, the invalid object isremoved from an object image, as shown in FIG. 6C.

When it is determined in operation S300 that occlusion of objects occurs(S300-YES), the multi-object tracker 150 removes the occlusion (S350)and determines whether hijacking occurs (S360).

In operation S350, the multi-object tracker 150 estimates a bounding boxand depth of each object and removes the occlusion by removing objectsexcept for the target object from the object image.

Specifically, the operation of removing the objects except for thetarget object from the object image by estimating the bounding box anddepth of each object may be a process in which a depth order of objectsis determined based on the bounding box and depth of each object, atleast one of objects in front of and behind the target object isrecognized on the basis of the determined depth order, and then therecognized object is removed.

FIG. 7 is a picture showing an example in which the multi-object trackerremoves occlusion of objects in the embodiment of the present invention.FIG. 7A shows an example in which occlusion of objects occurs in theembodiment of the present invention, and FIG. 7B shows an example inwhich the occlusion is removed in the embodiment of the presentinvention.

When the multi-object tracker removes occlusion of objects, regions(regions inside bounding boxes indicated by a dashed line) which aredetermined to be occluded by a target object are deleted, as shown inFIG. 7B.

In addition, in operation S360, when a displacement of the bounding boxof the target object is detected to be identical to a displacement ofthe bounding box of another object for a predetermined time period, themulti-object tracker 150 determines that hijacking occurs.

When it is determined in operation S360 that hijacking does not occur(S360-NO), the multi-object tracker 150 outputs an object-tracking imagecorrected through a removal of occlusion of objects (S370).

When it is determined in operation S360 that hijacking occurs(S360-YES), the multi-object tracker 150 removes the hijacking (S380)and outputs an object-tracking image corrected through a removal of boththe occlusion of objects and the hijacking (S390).

FIG. 8 is a table for comparing center location errors of the system fortracking multiple objects according to the embodiment of the presentinvention and a conventional object tracking system. FIG. 9 is a tablefor comparing success rates of the system for tracking multiple objectsaccording to the embodiment of the present invention and theconventional object tracking system. FIG. 10 is a table for comparingmulti-object tracking times of the system for tracking multiple objectsaccording to the embodiment of the present invention and theconventional object tracking system.

As can be seen from FIG. 8, when the multi-object tracking was performedusing the system for tracking multiple objects according to theembodiment of the present invention, the center location error wasimproved by about 602% on average.

In addition, as can be seen from FIG. 9, when the multi-object trackingwas performed using the system for tracking multiple objects accordingto the embodiment of the present invention, the success rate wasimproved by about 157%.

Also, as can be seen from FIG. 10, the system for tracking multipleobjects according to the embodiment of the present invention processedapproximately 3.66 frames per second while the conventional objecttracking system processed approximately 0.243 frame per second.Therefore, when the multi-object tracking is performed using the systemfor tracking multiple objects according to the embodiment of the presentinvention, the multi-object tracking can be performed about 15 timesfaster than using the conventional object tracking system.

According to the multi-object tracking technology suggested by thepresent invention, it is possible to remove occlusion of objects andhijacking which may occur when multiple objects are simultaneouslytracked.

In addition, since the multi-object tracking technology proposed by thepresent invention is based on removing a non-target object causing anerror while a target object to be tracked is retained, it is possible tosimply and efficiently remove occurrences of occlusion of objects andhijacking.

Further, when the multi-object tracking technology proposed by thepresent invention is applied, parameter tuning according to a trackingenvironment can be optimized and an excellent result thereof cane beexpected in an actual tracking environment.

Although components or elements of the embodiments of the presentinvention are described as being combined into one component or elementor as being operated in a combined manner, the present invention is notlimited to the embodiments. In other words, one or more of thecomponents may be selectively combined to be operated without departingfrom a scope of the present invention. Further, each of the componentsmay be implemented as a single independent hardware device, but some orall of the components may be selectively combined to be implemented as acomputer program which includes a program module that performs some orall of functions combined in one or a plurality of hardware devices.Further, such a computer program may be stored in a computer readablemedia such as a Universal Series Bus (USB) memory, a compact disk (CD),or a flash memory to be read and executed by a computer to implement theexemplary embodiment of the present invention. Recording media of thecomputer program may include magnetic recording media, optical recordingmedia, and carrier wave media.

In the meantime, while the system and method for tracking multipleobjects according to the present invention have been particularly shownand described with reference to the specific embodiments thereof, itwill be understood by those of ordinary skill in the art that the scopeof the present invention is not limited to the particular embodimentsand various alternatives, changes and modifications may be made thereinwithout departing from the technical idea and scope of the presentinvention.

Therefore, it would be understood that the embodiments disclosed in thepresent invention and the attached drawings are not to limit thetechnical idea of the present invention but to describe the presentinvention, and the technical and protective scope of the presentinvention shall not be defined by the illustrated embodiments and theattached drawings. It should be also understood that the protectivescope of the present invention is interpreted by the following claimsand all technical ideas within the equivalent scope belong to thetechnical scope of the present invention.

What is claimed is:
 1. A system for tracking multiple objects,comprising: an image capturing unit configured to capture a video of apredetermined observation area and output the captured video; and amulti-object tracker configured to output an object-tracking image bytracking multiple objects within an object image which is generated byextracting the objects from each of image frames obtained from the videoobtained from the image capturing unit, wherein the multi-object trackerdetermines whether occlusion of the objects or hijacking occurs whileperforming multi-object tracking, and when it is determined that atleast one of the occlusion and the hijacking occurs, the multi-objecttracker outputs the object-tracking image corrected by removing theoccurring occlusion or hijacking.
 2. The system of claim 1, wherein themulti-object tracker outputs a current object-tracking image when it isdetermined that the occlusion of the objects and the hijacking do notoccur.
 3. The system of claim 1, wherein, when a bounding box of atarget object is detected to be occluded by a bounding box of anotherobject, the multi-object tracker determines that the occlusion of theobjects occurs.
 4. The system of claim 1, wherein, when it is determinedthat the occlusion of the objects occurs, the multi-object trackerestimates a bounding box and depth of each of the objects and removesobjects except for a target object from the object image.
 5. The systemof claim 4, wherein the multi-object tracker determines a depth order ofthe objects on the basis of the estimated bounding boxes and depths ofthe objects, recognizes at least one of objects in front of and behindthe target object on the basis of the determined depth order of theobjects, and removes the recognized object.
 6. The system of claim 1,wherein, when a displacement of a bounding box of a target object isdetected to be identical to a displacement of a bounding box of anotherobject for a predetermined time period, the multi-object trackerdetermines that the hijacking occurs.
 7. The system of claim 1, wherein,when it is determined that the hijacking occurs, the multi-objecttracker removes an object being actually tracked by a bounding box of atarget object from the object image so that the bounding box tracks thetarget object.
 8. The system of claim 1, wherein the multi-objecttracker generates a reference background image by modeling a referencebackground using a reference background modeling video obtained from theimage capturing unit and generates the object image by extracting theobjects from each of the image frames on the basis of comparison betweenthe reference background image and each of the image frames obtainedfrom a video for object tracking obtained from the image capturing unit.9. The system of claim 8, wherein the multi-object tracker obtains acolor difference by comparing colors of the reference background imageand each of the image frames, and extracts the objects from each of theimage frames on the basis of the obtained color difference.
 10. Thesystem of claim 1, wherein, when the generated object image is aninitial object image, the multi-object tracker allocates a bounding boxto each of the objects to initialize multi-object tracking and performsthe multi-object tracking.
 11. A method of tracking multiple objects,comprising: generating a reference background image by modeling areference background using a reference background modeling videoobtained from an image capturing unit; generating an object image byextracting objects from each of image frames on the basis of comparisonbetween the reference background image and each of the image framesobtained from a video for object tracking obtained from the imagecapturing unit; and outputting a current object-tracking image or acorrected object-tracking image according to a result of determinationon whether occlusion of the objects or hijacking occurs, whileperforming multi-object tracking on the basis of the object image. 12.The method of claim 11, wherein whether the occlusion of the objectsoccurs is determined on the basis of detection of whether a bounding boxof a target object is occluded by a bounding box of another object. 13.The method of claim 11, wherein whether the hijacking occurs isdetermined on the basis of detection of whether a displacement of abounding box of a target object is identical to a displacement of abounding box of another object.
 14. The method of claim 11, wherein theoutputting of the current object-tracking image is performed when it isdetermined that the occlusion of the objects and the hijacking do notoccur.
 15. The method of claim 11, wherein, when it is determined thatthe occlusion of the objects occurs, the outputting of the correctedobject-tracking image is performed wherein the object-tracking image iscorrected by removing objects except for a target object from the objectimage by estimating a bounding box and depth of each of the objects. 16.The method of claim 11, wherein, when it is determined that thehijacking occurs, the outputting of the corrected object-tracking imageis performed wherein the object-tracking image is corrected by removingan object being actually tracked by a bounding box of a target objectfrom the object image so that the bounding box tracks the target object.17. The method of claim 11, wherein, when it is determined that theocclusion of the objects and the hijacking occur, the outputting of thecorrected object-tracking image is performed wherein the object-trackingimage is corrected by removing objects except for a target object fromthe object image by estimating a bounding box and depth of each of theobjects and is corrected also by removing an object being actuallytracked by a bounding box of the target object from the object image sothat the bounding box tracks the target object.
 18. The method of claim15, wherein the removing of the objects except for the target objectfrom the object image by estimating the bounding box and depth of eachof the objects includes determining a depth order of the objects on thebasis of the estimated bounding boxes and depths of the objects,recognizing at least one of objects in front of and behind the targetobject on the basis of the determined depth order, and removing therecognized object.
 19. The method of claim 11, wherein the generating ofthe object image includes obtaining a color difference by comparingcolors of the reference background image and each of the image framesand extracting the objects from each of the image frames on the basis ofthe obtained color difference.
 20. The method of claim 11, wherein, wheninitial multi-object tracking is performed, the performing of themulti-object tracking on the basis of the object image includesallocating a bounding box to each of the objects to initialize themulti-object tracking and then performing the multi-object tracking.